HETEROGENEOUS MULTI-SENSOR FUSION FOR 2D AND 3D POSE ESTIMATION

Sensor fusion is a process in which data from different sensors is combined to acquire an output that cannot be obtained from individual sensors. This dissertation first considers a 2D image level real world problem from rail industry and proposes a novel solution using sensor fusion, then proceeds further to the more complicated 3D problem of multi sensor fusion for UAV pose estimation. One of the most important safety-related tasks in the rail industry is an early detection of defective rolling stock components. Railway wheels and wheel bearings are two components prone to damage due to their interactions with the brakes and railway track, which makes them a high priority when rail industry investigates improvements to current detection processes. The main contribution of this dissertation in this area is development of a computer vision method for automatically detecting the defective wheels that can potentially become a replacement for the current manual inspection procedure. The algorithm fuses images taken by wayside thermal and vision cameras and uses the outcome for the wheel defect detection. As a byproduct, the process will also include a method for detecting hot bearings from the same images. We evaluate our algorithm using simulated and real data images from UPRR in North America and it will be shown in this dissertation that using sensor fusion techniques the accuracy of the malfunction detection can be improved. After the 2D application, the more complicated 3D application is addressed. Precise, robust and consistent localization is an important subject in many areas of science such as vision-based control, path planning, and SLAM. Each of different sensors employed to estimate the pose have their strengths and weaknesses. Sensor fusion is a known approach that combines the data measured by different sensors to achieve a more accurate or complete pose estimation and to cope with sensor outages. In this dissertation, a new approach to 3D pose estimation for a UAV in an unknown GPS-denied environment is presented. The proposed algorithm fuses the data from an IMU, a camera, and a 2D LiDAR to achieve accurate localization. Among the employed sensors, LiDAR has not received proper attention in the past; mostly because a 2D LiDAR can only provide pose estimation in its scanning plane and thus it cannot obtain full pose estimation in a 3D environment. A novel method is introduced in this research that enables us to employ a 2D LiDAR to improve the full 3D pose estimation accuracy acquired from an IMU and a camera. To the best of our knowledge 2D LiDAR has never been employed for 3D localization without a prior map and it is shown in this dissertation that our method can significantly improve the precision of the localization algorithm. The proposed approach is evaluated and justified by simulation and real world experiments

MeetingBank: A Benchmark Dataset for Meeting Summarization

As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings. However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential. Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference. This lack of annotated corpora has hindered the development of meeting summarization technology. In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade. MeetingBank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting. This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks. The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made. We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques. Our dataset can be accessed at: https://meetingbank.github.ioComment: ACL 2023 Long Pape

Boosting Punctuation Restoration with Data Generation and Reinforcement Learning

Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability. While punctuated texts are abundant from written documents, the discrepancy between written punctuated texts and ASR texts limits the usability of written texts in training punctuation restoration systems for ASR texts. This paper proposes a reinforcement learning method to exploit in-topic written texts and recent advances in large pre-trained generative language models to bridge this gap. The experiments show that our method achieves state-of-the-art performance on the ASR test set on two benchmark datasets for punctuation restoration.Comment: Accepted at INTERSPEECH 2023, 6 page

Sensor fused three-dimensional localization using IMU, camera and LiDAR

© 2016 IEEE. Estimating the position and orientation (pose) of a moving platform in a three-dimensional (3D) environment is of significant importance in many areas, such as robotics and sensing. In order to perform this task, one can employ single or multiple sensors. Multi-sensor fusion has been used to improve the accuracy of the estimation and to compensate for individual sensor deficiencies. Unlike the previous works in this area that use sensors with the ability of 3D localization to estimate the full pose of a platform (such as an unmanned aerial vehicle or drone), in this work we employ the data from a 2D light detection and ranging (LiDAR) sensor, which can only estimate the pose in a 2D plane. We fuse it in an extended Kalman filter with the data from camera and inertial sensors showing that, despite the incomplete estimation from the 2D LiDAR, the overall estimated 3D pose can be improved. We also compare this scenario with the case where the 2D LiDAR is replaced with a 3D LiDAR with similar characteristics, but the ability of complete 3D pose estimation

Fuzzy adaptive extended Kalman filter for robot 3D pose estimation

Purpose Estimating the pose – position and orientation – of a moving object such as a robot is a necessary task for many applications, e.g., robot navigation control, environment mapping, and medical applications such as robotic surgery. The purpose of this paper is to introduce a novel method to fuse the information from several available sensors in order to improve the estimated pose from any individual sensor and calculate a more accurate pose for the moving platform. Design/methodology/approach Pose estimation is usually done by collecting the data obtained from several sensors mounted on the object/platform and fusing the acquired information. Assuming that the robot is moving in a three-dimensional (3D) world, its location is completely defined by six degrees of freedom (6DOF): three angles and three position coordinates. Some 3D sensors, such as IMUs and cameras, have been widely used for 3D localization. Yet, there are other sensors, like 2D Light Detection And Ranging (LiDAR), which can give a very precise estimation in a 2D plane but they are not employed for 3D estimation since the sensor is unable to obtain the full 6DOF. However, in some applications there is a considerable amount of time in which the robot is almost moving on a plane during the time interval between two sensor readings; e.g., a ground vehicle moving on a flat surface or a drone flying at an almost constant altitude to collect visual data. In this paper a novel method using a “fuzzy inference system” is proposed that employs a 2D LiDAR in a 3D localization algorithm in order to improve the pose estimation accuracy. Findings The method determines the trajectory of the robot and the sensor reliability between two readings and based on this information defines the weight of the 2D sensor in the final fused pose by adjusting “extended Kalman filter” parameters. Simulation and real world experiments show that the pose estimation error can be significantly decreased using the proposed method. Originality/value To the best of the authors’ knowledge this is the first time that a 2D LiDAR has been employed to improve the 3D pose estimation in an unknown environment without any previous knowledge. Simulation and real world experiments show that the pose estimation error can be significantly decreased using the proposed method

AUTOMATIC METHOD FOR DETECTING AND CATEGORIZING TRAIN CAR WHEEL AND BEARING DEFECTS

ABSTRACT Worldwide, railways are among the safest transportation services. Nevertheless, every year some serious accidents are reported. A noticeable portion of these accidents are a result of defective wheels, bearings, or brakes. Train wheels are subjected to different types of damage due to their interaction with the brakes and the track and they are required to be periodically inspected to ensure they meet all the safety criteria for proper operation. If the wheel damage remains undetected, it can worsen and result in overheating and severe damage to the wheel and track. There are a variety of wheel damages, classified in different groups based on the type and severity of the defect. The most usual cause of damage is severe braking, which applies directly to the wheel and results in local heating of the wheel. This can stop the wheel from rotating while the train is still moving, producing a defect called a "flat spot" or "hot spot". Flat-spotted wheels are a serious concern for the railroad industry. Depending on the level of wheel defect, different solutions should be taken. This paper will focus on automatically detecting flat-spotted wheels from thermal imagery using computer vision methods and introduces an algorithm to detect hot bearings. We first extract and segment both the wheel and bearing regions from the whole image, then we introduce a fuzzy inference to categorize the level of wheel damage. The whole process is done automatically and without any need for time consuming and costly manned inspection. Based on the severity of the defect, it can be decided which solution should be taken. INTRODUCTION Train wheels are made of steel, shaped carefully and heat treated during the manufacturing process to have a specific hardness. They are subjected to wear and different types of defects because of their interaction with the brakes and track. A defective wheel can cause damage to both track and the vehicle. If the defect is detected early, the wheel can be repaired or replaced before any derailment, accident or further damage happens. In order to identify such damage at an early stage, there are two choices: first is costly manned inspection, which is extremely time consuming and still does not guarantee 100 percent precision. Furthermore, the defects may have not happened by the time of inspection. Remote automatic detection is an alternative option. The second option is definitely faster and cheaper and hence can be done more often. For the remainder of this paper, different types of wheel defects will be named with an explanation of their causes, then different sensors that are used for wheel and bearing inspection will be introduced and, at the end, we introduce our proposed algorithm for flat-spotted wheel and also hot bearing detection which increases the maintenance efficiency

Heterogeneous Multisensor Fusion for Mobile Platform Three-Dimensional Pose Estimation

Copyright © 2017 by ASME. Precise, robust, and consistent localization is an important subject in many areas of science such as vision-based control, path planning, and simultaneous localization and mapping (SLAM). To estimate the pose of a platform, sensors such as inertial measurement units (IMUs), global positioning system (GPS), and cameras are commonly employed. Each of these sensors has their strengths and weaknesses. Sensor fusion is a known approach that combines the data measured by different sensors to achieve a more accurate or complete pose estimation and to cope with sensor outages. In this paper, a three-dimensional (3D) pose estimation algorithm is presented for a unmanned aerial vehicle (UAV) in an unknown GPS-denied environment. A UAV can be fully localized by three position coordinates and three orientation angles. The proposed algorithm fuses the data from an IMU, a camera, and a two-dimensional (2D) light detection and ranging (LiDAR) using extended Kalman filter (EKF) to achieve accurate localization. Among the employed sensors, LiDAR has not received proper attention in the past; mostly because a two-dimensional (2D) LiDAR can only provide pose estimation in its scanning plane, and thus, it cannot obtain a full pose estimation in a 3D environment. A novel method is introduced in this paper that employs a 2D LiDAR to improve the full 3D pose estimation accuracy acquired from an IMU and a camera, and it is shown that this method can significantly improve the precision of the localization algorithm. The proposed approach is evaluated and justified by simulation and real world experiments

Automatic method for detecting and categorizing train car wheel and bearing defects

Worldwide, railways are among the safest transportation services. Nevertheless, every year some serious accidents are reported. A noticeable portion of these accidents are a result of defective wheels, bearings, or brakes. Train wheels are subjected to different types of damage due to their interaction with the brakes and the track and they are required to be periodically inspected to ensure they meet all the safety criteria for proper operation. If the wheel damage remains undetected, it can worsen and result in overheating and severe damage to the wheel and track. There are a variety of wheel damages, classified in different groups based on the type and severity of the defect. The most usual cause of damage is severe braking, which applies directly to the wheel and results in local heating of the wheel. This can stop the wheel from rotating while the train is still moving, producing a defect called a “flat spot” or “hot spot”. Flat-spotted wheels are a serious concern for the railroad industry. Depending on the level of wheel defect, different solutions should be taken. This paper will focus on automatically detecting flat-spotted wheels from thermal imagery using computer vision methods and introduces an algorithm to detect hot bearings. We first extract and segment both the wheel and bearing regions from the whole image, then we introduce a fuzzy inference to categorize the level of wheel damage. The whole process is done automatically and without any need for time consuming and costly manned inspection. Based on the severity of the defect, it can be decided which solution should be taken

Detection of sliding wheels and hot bearings using wayside thermal cameras

Copyright © 2016 by ASME. Train car wheels are subjected to different types of damages due to their interactions with the brake shoes and track. If not detected early, these defects can worsen, possibly causing damage to the bogie and rail. In the worst-case scenario, this rail damage can possibly lead to later derailments, a serious concern for the rail industry. Therefore, automatic inspection and detection of wheel defects are high priority research areas. An automatic detection system not only can prevent train and rail damage, but also can reduce operating costs as an alternative for tedious and expensive manned inspection. The main contribution of this paper is to develop a computer vision method for automatically detecting the defects of rail car wheels using a wayside thermal camera. We concentrate on identification of flat-spotted/sliding wheels, which is an important issue for both wheel and suspension hardware and also rail and track structure. Flat spots occur when a wheel locks up and slides while the vehicle is still moving. As a consequence, this process heats up local areas on the metal wheel, which can be observed and potentially detected in thermal imagery. Excessive heat buildup at the flat spot will eventually lead to additional wheel and possibly rail damage, reducing the life of other train wheels and suspension components, such as bearings. Furthermore, as a byproduct of our algorithm, we propose a method for detecting hot bearings. A major part of our proposed hot bearing detection algorithm is common with our sliding wheel detection algorithm. In this paper, we first propose an automatic detection and segmentation method that identifies the wheel and bearing portion of the image. We then develop a computer vision method, using Histogram of Oriented Gradients to extract features of these regions. These feature descriptors are input to a Support Vector Machine classifier, a fast classifier with a good detection rate, which can detect abnormalities in the wheel. We demonstrate our methods on several real data sets taken on a Union Pacific rail line, identifying sliding wheels and hot bearings in these images

Sensor fusion of wayside visible and thermal imagery for rail car wheel and bearing damage detection

Two major components of rolling stock that are always of great interest when it comes to maintenance and safety related issues are car wheels and bearings. Rail car wheels are subjected to a variety of damage types due to their interaction with the track and brakes. It is important for the rail industry to detect these defects and take proper action at an early stage, before more damage can be caused to the train or possibly the track and to prevent possible safety hazards. Different inspection sensors and systems, such as wheel impact monitors, wheel profile detectors, hotbox detectors and acoustic detection technologies, are employed to detect different types of wheel and bearing defects. Usually no single sensor can accurately detect all kinds of damages and hence a combination of different sensors and systems and manual inspection by experts is used for wheel maintenance purposes and to guarantee train safety. The more complete and accurate the automatic defect detections are, the less manual examination is necessary, leading to potential savings in inspection time/resources and rail car maintenance costs. Wayside thermal and visible spectrum cameras are one option for the automatic wheel and bearing inspection. Each of these sensors has their own strengths and weaknesses. There are some types of defects that are not detectable at an early stage in the images taken by a vision camera, however these defects generate a distinctive heat pattern on the wheel or bearing that is clearly visible in the thermal imagery. On the other hand, other damages might be detectable from the visible spectrum image, but not necessarily have a distinguishable heat pattern in the thermal imagery. Since a thermal image is basically built of solely temperature data, it excludes other critical information, such as texture or color. This makes thermal and visible spectrum imagery complementary and if the images are fused the result will benefit from the strengths of both sensors. In this paper, wavelet decomposition is employed to extract the features of the thermal and vision imagery. Then the two images are merged based on their decompositions and a fused image is composed. The resulting fused image contains more information than each individual image and can be used as an input for image-based wheel and bearing defect detection algorithms. To verify the proposed method and to show an example of this application, it is demonstrated on a real data set from a Union Pacific rail line to identify sliding wheels